All Specters can be physically touched by your character, however, all weapons other than the Sigil are supposed to pass through the Specter (The Specter does not take splash damage from those weapons) .
In Vavoom, all weapons other than the Sigil seem to hit the Specter, even though they probably do no damage.